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Abstract 


Summer learning loss can have a major and cumulative 
impact on academic achievement and disproportionately 
affect children from low-income households, who often 
have less access to learning resources over the summer. 
Approximately two-thirds of the ninth-grade reading 
achievement gap between children from high- and 
low-income backgrounds could be attributed to summer 
learning loss in the first five years of school." 


The purpose of this study was to test the impact of 
an online curriculum, ABCmouse.com Early Learning 
Academy, on “summer slide”: the loss of previously 
learned skills in reading and mathematics.** A diverse 
sample of 999 rising first graders enrolled in Miami-Dade 
County Public Schools (M-DCPS) participated in the study. 
Children who were randomly assigned to the treatment 
group were given free home access to ABCmouse over 
the summer with a weekly minimum usage requirement 
of 45-60 minutes and 15 ABCmouse Learning Activities. 
Children who were randomly assigned to the control 
group were not given ABCmouse. To examine summer 
slide, we compared district-provided i-Ready assessment 
data in reading and math from spring 2016 and fall 2016, 
and we evaluated subgroups based on spring 2016 
Stanford Achievement Test-10 (SAT-10) scores in reading 
and math. 


ABCmouse helped prevent summer slide and contributed 
to a net gain in reading for children who completed 
at least 208 Learning Activities over the 12 weeks 


* Associate Professor of Psychology, University of Miami. 


of summer (approximately 17 Learning Activities or 
70 minutes of ABCmouse use per week), compared to 
the control group. The subset of treatment group children 
who met this usage threshold demonstrated significantly 
greater gains—approximating the benefits of one month 
of instructional time—during the summer, compared to 
the control group children (p < .05, Cohen’s d = 013). 
Importantly, these children showed sustained benefits 
from using ABCmouse, approximating two more months 
of academic instruction than the control group by winter 
of the following academic year (p < .01, d = 0.18). 


Participants 


In summer 2016, 1,214 children were consented by their 
parents or guardians to participate in the study from the 
entire M-DCPS population of 23,490 rising first graders. 
Eligibility criteria for the study were that participants must 
be enrolled in an M-DCPS kindergarten class in spring 
2016 and have access to the internet and a computer or 
mobile device for regular weekly use over the summer. 
This sample (49% girls, 51% boys) was predominantly 
Hispanic (66%), 23% Black, 8.5% White, 1% Asian, and 
1% Mixed or other, and 59% of the sample were enrolled in 
Title | schools during the spring of kindergarten.? Across 
all M-DCPS elementary schools, 78.4% of children were 
eligible for a free/reduced lunch during the 2015—2016 
school year. 


Of the consented sample, 789 children were randomly 
assigned to the treatment group,’ with 608 families 


** M.S. in Developmental Psychology, University of Miami; Ph.D. candidate, expected 2019. 


"This research was funded by Age of Learning, Inc. The authors were responsible for and had full control over study design, research findings, and this report. 


1 Alexander, K. L., Entwisle, D. R., & Olson, L. S. (2007). Lasting consequences of the summer learning gap. American Sociological Review, 72(2), 167—180. 


2 More than 50% of parents who completed the end-of-study survey reported household annual income of $50,000 or less, and another 15% reported 


$50,000-75,000. 





3 Two-thirds of the children were assigned to the treatment group and one-third were assigned to the control group. In light of projected attrition, over-enrollment 


into the treatment group was done to ensure a sufficient sample size in the treatment group for analysis at the end of the study. 
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redeeming their ABCmouse codes, and 426 were 
randomly assigned to the control group, with 11 families 
subsequently deciding not to participate. After a search 
of the ABCmouse database that was performed at the 
end of summer, 24 children in the control group were 
identified as having some summer usage of the online 
program (completing at least one Learning Activity) and 
were excluded from the final analytic sample. 


There were no statistically significant differences across 
any demographic variables in the final analytic sample 
(N = 999), the consented study sample (N = 1,214), and the 
overall M-DCPS population (N = 23,490). The consented 
sample’s SAT-10 reading and math scores were slightly 
but significantly higher than the average M-DCPS 
kindergarten scores in spring 2016 (p < .05). In the final 
analytic sample, initial mean spring SAT-10 reading 
scores (p < .01) and spring SAT-10 math scores (p < .05) 
were slightly but statistically significantly higher in the 
treatment group compared to the control group (reading: 
treatment M = 534.98, SD = 53.75 vs. control M = 522.82, 
SD = 57.07; math: treatment M = 531.45, SD = 44.87; 
control M = 524.17, SD = 44.07). However, differences in 
spring i-Ready scores, the primary outcome measure, 
were not statistically different among the final analytic 
sample, the consented sample, and the overall M-DCPS 
population (p’s > 10). 


Design and Procedure 


Children in the treatment group were provided with 
free home access to ABCmouse over the summer 
with a weekly minimum usage requirement of 45—60 
minutes and 15 Learning Activities, preferably at Level 
7 (early first-grade content). Parents received weekly 
feedback on their children’s usage via email. If usage fell 
below recommended levels, a reminder to encourage 
children’s usage was included in the email. In addition, 
parents in both the control and treatment groups 
completed brief online technology diaries each week 
to report their children's use of technology. At the end 
of the summer, all parents were asked to complete a 
20-minute online survey regarding their views on the use 
of educational technology and the impact of technology 


on their children’s learning over the summer. All parent 
correspondence and surveys were available in Spanish. 
Parents received a $20 gift card for their participation 
in the end-of-summer survey. Control group children 
received free home access to ABCmouse for three 
months after fall i-Ready assessments were completed 
(September 30, 2016). 


i-Ready assessment data served as the primary 
outcome measure for the study, and SAT-10 scores, a 
comprehensive and nationally standardized measure, 
provided descriptive information on the achievement 
levels of the study samples to create subgroups of 
children (high scorers and low scorers) for analyses. 


Overall, the 608 children who redeemed their ABCmouse 
codes spent an average of 985 minutes on ABCmouse 
(range = 0—10,523 minutes; SD = 1,082; M = 82 minutes 
per week), and completed on average a total of 
203 ABCmouse Learning Activities (range = O-2,472; 
SD = 246; M = 16.9 per week) from June 1, 2016 through 
August 22, 2016. Ofthose activities completed, an average 
of 158 (78%) were Level 7 activities (range = O-1,270; 
SD = 196). Overall, participants completed approximately 
three times more reading activities during the summer 
(Mean = 128.8, Median = 74) than math activities 
(Mean = 44.6, Median = 19). 


M-DCPS assessment data for the entire rising first-grade 
population revealed that a large percentage of children 
experienced a slide in their fall i-Ready scores; 44% of 
entering first graders had summer reading loss and 54% 
had math loss. On average, rising M-DCPS first graders 
gained 2.26 points on i-Ready reading scores and lost 
3.35 points on i-Ready math scores from spring 2016 
to fall 2016. For those children who slid in reading, the 
median loss was 16 points; for those children who slid in 
math, the median loss was 14 points. A loss of 4—5 points 
on i-Ready is approximately equivalent to one month 
of instruction during a 9- or 10-month academic year 
(30 weeks of instruction). National studies find that, on 
average, children have a loss of about one month of 
instruction over the summer. 


4 RAND (2011). Making summer count: How summer programs can boost children’s learning. RAND, Inc., http://www.rand.org. 
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Because the spring i-Ready assessment was not given 
at the very end of the school year and the fall i-Ready 
assessment was not given at the very beginning of the 
next school year, these scores provide only a rough 
measure of summer slide. There could have been 
meaningful learning gains from school instruction that 
occurred in the final weeks of the spring semester and/or 


Results 


the first few weeks of the fall semester, “dampening” 
the measurable effects of summer slide. In addition, it 
is possible that children in the control group accessed 
ABCmouse during the prior kindergarten year and/or over 
the summer in public libraries, through summer learning 
programs, or at home through ABCmouse accounts that 
we were not able to identify. 


Finding 1. ABCmouse helped prevent summer slide in reading for children with regular usage compared to the 


control group. 


ABCmouse helped prevent summer slide in reading for children who completed at least 208 Learning Activities over the 
12 weeks of summer (an average of 17 Learning Activities and 70 minutes of ABCmouse usage per week), as compared 


to the control group. As seen in Figure 1, children who met this usage threshold (“regular usage”) showed significantly 


greater gains over the summer compared to the control group, an average of 3.8 points more growth, approximating 


the benefits of one month of academic instruction.° Multilevel regression models controlling for spring pretest reading 


scores, age, gender, race, and Spanish language at home confirmed this finding, with predicted score differences 
between groups of 4.7 points (B = 4.7, p < .05, Cohen’s d = 0.13).° 


Summer Reading Gains 
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Figure 1. Gains in reading by treatment group children who completed at least 208 ABCmouse 
Learning Activities over the summer and control group children who did not use ABCmouse (N = 207 for 


treatment group and N = 391 for control group). 


5 There were no significant differences in spring scores between this smaller, higher-usage group and the overall treatment group. 


® Not all children completed both spring and fall assessments. Any missing data points were dealt with in the multilevel models through the use of Full 
Information Maximum Likelihood (FIML) estimation, a preferred method for handling missing data that uses all available data for each case when estimating 
parameters and is appropriate for handling the level of missing data in this study (Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full 
information maximum likelihood estimation for missing data in structural equation models. Structural equation modeling, 8(3), 430—457; McCartney, K., 
Burchinal, M. R., & Bub, K. L. (2006). Introduction to the monograph. Monographs of the Society for Research in Child Development, 71(3), 1-8). 
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In addition, treatment group children in the higher usage group were less likely to experience a summer slide of one 
month or more (greater than 5-point loss) in reading (control = 40%, treatment/regular usage = 33%).’ 


An intent-to-treat analysis was conducted of all children randomly assigned to the treatment group (N = 789) or to the 
control group (N = 408).° Overall, treatment group children showed a trend toward higher i-Ready fall reading scores 
compared to the control group, controlling for spring i-Ready reading scores and demographic variables (M treatment 
group = 41710, M control group = 414.91, p = .23, d = 0.05). An analysis including only treatment group children who 
redeemed their ABCmouse codes (N = 608) and control group children who did not use ABCmouse (N = 391) found that 
the treatment group children showed a stronger trend toward greater gains in reading compared to the control group 
(M treatment group = 418.34, M control group = 414.07, p = .08, d = 0.10). 


In the regular usage group, a lower percentage of children who scored below the median of the M-DCPS population at 
the end of kindergarten (M-DCPS population spring 2016 median SAT reading score = 509, median math score = 515) 
slid in reading than comparable low-scoring children in the control group (40% treatment vs. 48% control). However, this 
finding was not statistically significant in the regression analyses. 


Finding 2. The more children used ABCmouse over the summer, the greater their learning gains. 


A multiple regression analysis confirmed that the total number of ABCmouse Learning Activities completed over the 
summer positively contributed to gains in reading scores, controlling for spring pretest reading scores, age, gender, 
race, and Spanish language at home. For every 100 ABCmouse Learning Activities completed, we can expect a 1.1 point 
gain on i-Ready reading, which is approximately equivalent to one week of academic instruction, controlling for the 
effect of prior knowledge and demographic factors (B = .011, p < .05). 


In light of the fact that children completed relatively few math activities on ABCmouse (less than 20% of the total 
number of Learning Activities completed), it was not surprising that no differences were found in math outcomes 
between the treatment and control groups. However, within the treatment group, a higher level of usage was associated 
with positive trends in math gains. Despite significantly fewer completed math activities than reading activities, the 
difference in math scores between treatment group children who used ABCmouse the most and treatment group 
children who used ABCmouse the least indicated a meaningful positive trend associating ABCmouse usage with math 
learning, and approached significance: 3.2 points higher or approximating the benefit of several weeks of academic 
instruction (p = .08, d = 0.16; N = 124 for higher math users and 379 for lower math users). On average, children in the top 





quartile of usage over the summer completed 44 total math activities. This positive trend in math might have reached 
Statistical significance had children completed more math activities during the study; there was also a steeper “slide” in 
math overall. 


Finding 3. ABCmouse’s positive impact on reading scores was sustained through the following winter. 


The benefits of regular ABCmouse usage over the summer were sustained through at least the first half of the following 
academic year. Using similar multiple regression models, controlling for spring 2016 i-Ready reading scores and 
demographic variables, children who completed a total of at least 208 activities over the summer continued to show higher 
scores, on average 8.03 points in i-Ready reading at the second assessment period (November 28—December 23, 2016), 
a benefit of approximately two months of academic instruction relative to the control group (B = 8.03, p < .01, d = 0.18; 
M regular user group = 451.80, N = 165; M control group = 443.54, N = 323). Those children who met a threshold of 
168 activities completed over the course of the summer months (averaging 14 Learning Activities per week over 12 weeks) 


7 50% of the control group children experienced some “slide” in reading compared to 44% of the treatment group children. 


An additional seven children in the control group were excluded from this analysis because they were either no longer enrolled in M-DCPS or they were not 
yet enrolled in M-DCPS and were therefore without assessment data 
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also showed sustained gains, scoring on average 6.36 points higher in i-Ready reading in winter, an approximate benefit 
of 1.3-1.6 months of academic instruction relative to the control group (B = 6.36, p = .02, d = 0.20; M regular user 
group = 449.90, N = 209; M control group = 443.54, N = 323). Spring 2017 i-Ready data was not provided by the district. 
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Figure 2. i-Ready winter reading scores for treatment group children in the regular usage group 
compared to control group children, controlling for spring scores and child demographics (N control 


group = 323, N regular usage group = 165). 


Finding 4. A large majority of parents reported that ABCmouse had a meaningful impact on their children’s 
learning and that they support the use of ABCmouse in first grade. 


In a survey completed at the end of the summer, 82% of the treatment group parents reported that ABCmouse had a 
meaningful impact on their children’s overall learning, and 89% said they would support the use of ABCmouse during 
first grade. The majority (81%) were very likely or likely to recommend ABCmouse to parents of other children to help 


them prepare for first grade. 


Conclusion 


Overall, rising first-grade children in M-DCPS experienced 
a moderate amount of summer loss in reading and 
math during the summer of 2016. However, children in 
the treatment group with regular usage of ABCmouse 
experienced less summer slide and a significant net gain 
in reading as compared to the control group. These gains 
approximated the benefits of one month of instructional 
time. Gains were sustained through at least the first half 


of the following academic year. Children in the treatment 
group benefited more, approximating two more months 
of academic instruction by winter, from their regular use 
of ABCmouse during the summer months. The survey 
results indicate that the majority of treatment group 
parents support the use of ABCmouse for their first-grade 
children, and treatment group parents specifically 
recommend the use of ABCmouse to other parents. 
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